Systems and methods for simulating sense data and creating perceptions

ABSTRACT

A method includes defining a first virtual being (e.g., including sensory locations for sensors, sense locations for sense properties, artificial neural networks connecting sensors to sense properties) in a virtual environment. The method also includes defining an object (e.g., including sense locations) in the virtual environment. The method also includes, in accordance with an interaction between the virtual being and the object, receiving sensory input at a first sensor at a first sensory location using a first virtual medium according to a first sense property of the object at a first sense location. The first sensor, the first virtual medium, and the first sense property have a same sensory type. According to the received sensory input, a first artificial neural network translates the received sensory input into updates to one or more configuration parameters of sensors of the first virtual being or movement of the virtual being.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/997,867, filed Aug. 19, 2020, entitled “Systems and Methods forSimulating Sense Data and Creating Perceptions,” which claims priorityto U.S. Provisional Patent Application No. 62/889,063, filed Aug. 20,2019, entitled “Method of simulating sense data and creating perceptionsusing an enhanced virtual body and enhanced virtual environment,” eachof which is incorporated by reference in its entirety.

TECHNICAL FIELD

The disclosed implementations relate generally to virtual reality andmore specifically to simulating sense data and creating perceptions.

BACKGROUND

Conventional automatons rely on real-world sensory data for generatingresponses. Such machines are capable of acting independently, but arereactive and lack awareness. For example, robots and semi-autonomous orautonomous vehicles use sensory data from cameras, microphones,satellite-based radio, Lidar, and other electronic sensors and systems,for navigation. Some robots use cameras and microphones to interact withpeople, and use mechanical actuators to create facial expressions.Reaction alone is not awareness, as it involves no perception, only ameasurement and an instruction. For example, a robot arm with a heatsensor may have an instruction to retract the arm when the temperatureread by the sensor is above a certain temperature.

State-of-the-art robots and autonomous systems that incorporate advancedartificial intelligence (AI) algorithms lack experiences and associatedcapabilities. For example, an AI system lacks mastery over language,because quality of such language capabilities is correlated withrichness of experience. To have a rich set of experiences across aspectrum requires such AI-based systems to have a body with a large setof senses and an environment that closely approximates the real world.However, inserting the AI into a mammal body is impractical.

SUMMARY

Accordingly, there is a need for simulating senses in a virtual body ina virtual environment, to create perceptions. Systems, devices, andmethods described herein may be used to simulate (or approximate) sensedata and create perceptions using an enhanced virtual body and enhancedvirtual environment. Some implementations use fundamental qualityrepresentation and recreation algorithms (FQRRs), a model of pain and/orpleasure, and apply machine learning algorithms as an AI brain for theenhanced virtual body. In some implementations, the AI brain utilizesone or more machine learning libraries and frameworks (e.g., Google'sTensorFlow or similar machine learning or AI frameworks provided by AWS,Google's Machine Learning (ML) Kit, Microsoft Azure AI, OpenAI API,Apple Core ML, PyTorch).

In accordance with some implementations, a method is provided forsimulating sense data for virtual beings in virtual environments. Themethod is performed at a computer system having one or more processors,memory, and one or more programs stored in the memory and configured forexecution by the one or more processors. The one or more programsinstructions for performing the method. The method includes defining afirst virtual being in a virtual environment. The first virtual beingincludes a plurality of sensory locations. Each sensory location has oneor more sensors. Each sensor has a respective sensory type and isconfigured to receive sensory input by one or more respective virtualmediums having the respective sensory type. The first virtual being alsoincludes a plurality of sense locations, each sense location storing arespective set of one or more sense properties, each sense propertyhaving a respective sensory type. The first virtual being also includesa plurality of artificial neural networks connecting sensors at theplurality of sensory locations. The method also includes defining anobject in the virtual environment. The object has a plurality of senselocations, each sense location storing a respective set of one or moresense properties, each sense property having a respective sensory type.The method also includes, in accordance with an interaction between thevirtual being and the object, receiving sensory input at a first sensorat a first sensory location using a first virtual medium according to afirst sense property of the object at a first sense location. The firstsensor, the first virtual medium, and the first sense property have asame sensory type. The method also includes, in accordance with thereceived sensory input, using a first artificial neural network totranslate the received sensory input into updates to one or moreconfiguration parameters of sensors of the first virtual being ormovement of the virtual being. The artificial neural network or thevirtual medium or combination can update the configuration parameters(e.g. resolution or pruning of sensors when their resolution or maxresolution goes to 0 in for some sensory types where it is a permanentpain effect that we want. The artificial neural network also can also beresponsible for actuating the movement of the being.

In some implementations, the method further includes: selectingenhancements or impairments to apply to the first sensory locationand/or the first artificial neural network based on (i) the firstsensory location and the first sense location, and (ii) the sensory typeof the first sensor, the first virtual medium, and the first senseproperty; and altering, based on the enhancements or impairments, (i)one or more configuration parameters of the first sensor, therebymodifying its ability to receive sensory input, and/or (ii) one or moreconfiguration parameters of the first artificial neural network, therebymodifying its ability to process sense properties.

In some implementations, in accordance with a determination that thesensory type is light touch, selecting the enhancements or impairmentsis further based on determining whether the first sensory location isdeformed by a predetermined amount in a direction normal to the firstsensory location's non-deformed baseline location, within apredetermined time period. Note that “light touch” is a label thatidentifies a tiny amount of deformation. An alternative label, such as“soft touch” could be used. The meaning is qualified based on having atiny amount of deformation.

In some implementations, altering the one or more configurationparameters of the first sensor includes enhancing the ability of thefirst sensor to receive sensory input at the first sensory location byincreasing resolution of the first sensor by a predetermined amount.

In some implementations, the method further includes: in accordance witha determination that the sensory type is temperature, selecting theenhancements or impairments further based on determining whether adistance between the first sensory location and the first sense locationis less than a predetermined amount in a direction normal to the firstsensory location's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes enhancing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by increasing resolution of the first sensor by a firstpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a predetermined minimum temperature up to apredetermined maximum temperature.

In some implementations, selecting the enhancements or impairments isfurther based on determining if the first sensory location is deformedby a predetermined amount in a direction normal to the first sensorylocation's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes enhancing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by increasing resolution of the first sensor by a secondpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a predetermined minimum temperature up to apredetermined maximum temperature.

In some implementations, the method further includes: after altering theone or more configuration parameters of the first sensor and/or the oneor more configuration parameters of the first artificial neural network:in accordance with a determination that the ability of the first sensorto receive sensory input, and/or the ability of the first artificialneural network to sense properties has not changed within apredetermined time period, impairing the ability of one or more sensorsto receive sensory input by decreasing resolution of the first sensor bya predetermined amount. The one or more sensors correspond to sensorylocations with a sensory type of light touch.

In some implementations, in accordance with a determination that thesensory type is pressure, selecting the enhancements or impairments isfurther based on determining whether the first sensory location isdeformed by greater than a predetermined amount in a direction normal tothe first sensory location's non-deformed baseline location, within apredetermined time period.

In some implementations, altering the one or more configurationparameters of the first sensor includes impairing the ability of thefirst sensor to receive sensory input at the first sensory location bydecreasing resolution of one or more sensors, at or near the firstsensory, for a predetermined time period. The one or more sensorscorrespond to sensory locations with a sensory type of light touch orpressure.

In some implementations, the method further includes, in accordance witha determination that resolution of a subset of sensors of the one ormore sensors equals 0, removing the subset of sensors from the firstvirtual being.

In some implementations, in accordance with a determination that thesensory type is temperature, selecting the enhancements or impairmentsis further based on determining whether a distance between the firstsensory location and the first sense location is less than apredetermined amount in a direction normal to the first sensorylocation's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes impairing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by decreasing resolution of the first sensor by a secondpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a first predetermined minimum temperature or below asecond predetermined maximum temperature.

In some implementations, in accordance with a determination that thesensory type is temperature, selecting the enhancements or impairmentsis further based on determining whether the first sensory location isdeformed by a predetermined amount in a direction normal to the firstsensory location's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes impairing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by decreasing resolution of the first sensor by a thirdpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a predetermined minimum temperature or below afourth predetermined maximum temperature.

In some implementations, the method further includes displaying thefirst virtual being on one or more displays of the computer system.

In some implementations, the object is a second virtual being, distinctfrom the first virtual being. In some implementations there are three ormore virtual beings, each with its own sensory locations, senselocations, and artificial neural networks.

In some implementations, each sensory location is a point, aone-dimensional segment, a two dimensional area, or a three dimensionalregion.

In some implementations, each sensory location is an n-dimensionalmanifold in the virtual environment, with n=0, 1, 2, or 3.

In some implementations, each sense location is an n-dimensionalmanifold in the virtual environment, with n=0, 1, 2, or 3.

In some implementations, each sensory location corresponds to arespective region, surface, or point, on or within the first virtualbeing.

In some implementations, the plurality of sensory locations aredynamically generated and associated with specific points and/or areason the first virtual being when a surface topology of the first virtualbeing is changing. However, sense properties usually remain constant intheir manifold relative positions.

In some implementations, the sensory type includes one or more of:temperature, light touch, pressure, vibration, stretch/compress, sound,and bright.

In some implementations, the method further includes: providing one ormore Application Programming Interface (API) calls to update theplurality of sensory locations; and, in response to receiving a call tothe one or more Application Programming Interface (API) calls,performing one or more operations selected from the group consisting of:creating, writing, reading, modifying, moving, and/or deleting a sensorylocation.

In some implementations, the first virtual being comprises a virtualapproximation of a body of human, an animal, an insect, a humanoid, or acreature.

In accordance with some implementations, a system for simulating sensedata for virtual beings in virtual environments includes one or moreprocessors, memory, and one or more programs stored in the memory. Theprograms are configured for execution by the one or more processors. Theprograms include instructions for performing any of the methodsdescribed herein.

In accordance with some implementations, a non-transitory computerreadable storage medium stores one or more programs configured forexecution by a computer system having one or more processors and memory.The one or more programs include instructions for performing any of themethods described herein.

Thus methods, systems, and graphical user interfaces are provided forinteractive visual analysis of a data set.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the aforementioned implementations of theinvention as well as additional implementations, reference should bemade to the Description of Implementations below, in conjunction withthe following drawings in which like reference numerals refer tocorresponding parts throughout the figures.

FIG. 1A illustrates conceptually a process of simulating sense data forvirtual beings in virtual environments in accordance with someimplementations.

FIG. 1B illustrates a publish-subscribe model of a process of simulatingsense data for virtual beings in virtual environments in accordance withsome implementations.

FIG. 2 is a block diagram of a computing device according to someimplementations.

FIGS. 3A, 3B, 3C, and 3D show an example of simulating sense data forvirtual beings in virtual environments in accordance with someimplementations.

FIGS. 4A, 4B, and 4C show an example of simulating human visionaccording to some implementations.

FIG. 5 illustrates example modulation effects, in accordance with someimplementations.

FIG. 6 provides a flowchart of a process for simulating sense data,according to some implementations.

Like reference numerals refer to corresponding parts throughout thedrawings.

Reference will now be made in detail to implementations, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the present invention. However,it will be apparent to one of ordinary skill in the art that the presentinvention may be practiced without these specific details.

DESCRIPTION OF IMPLEMENTATIONS

FIG. 1A illustrates conceptually a process 100 of simulating sense datafor virtual beings in virtual environments in accordance with someimplementations. Some implementations define a virtual being 110, one ormore objects and other virtual beings 112, and/or non-objects 114 (e.g.,area/plane, space/volume, or points), in a virtual environment. Thevirtual being 110, one or more objects and other virtual beings 112,and/or non-objects 114 are each associated (as indicated by the lines124, 126, and 128, respectively) with sense properties 108. Points ofobservation 104 (e.g., sensors) correspond to sensory locations on thevirtual being 110. Locations (or places) of the virtual being 110, oneor more objects and other virtual beings 112, and/or non-objects 114,correspond to regions, surfaces, or points, on or in the virtual being110, one or more objects and other virtual beings 112, and/ornon-objects 114, respectively. When the virtual being 110 interacts withthe one or more objects and other virtual beings 112, a virtual medium106 (sometimes called a virtual medium of observation, or a simulatedmedium of observation) computes what is sensed, based on location andtype of the sense properties 108 and points of observation 104. Forexample, the virtual medium computes/incorporates distance or proximity,contact and/or soft-body deformation between the sense properties 108and points of observation 104, according to some implementations. Thesense properties 108, virtual medium 106, and points of observation 104each have associated type descriptors (sometimes called sensory types;e.g., temperature). The sensory type or type descriptor is used to matchor trigger the corresponding sense properties 108, virtual medium 106,and points of observation 104, when the virtual being 110 interacts withthe one or more objects and other virtual beings 112. In someimplementations, the virtual medium 106 sets a value of the points ofobservation 104 in accordance with the interaction. The value can be aclosure, or reference or pointer to a programming language type instancethat, as an effect, modulates the points of observation 104 on thevirtual being 110 (or the body of the virtual being 110). In someimplementations, the value is passed along to an Artificial Intelligence(AI) or sense data stream processing 102 (sometimes called AI and/orstream processing module; e.g., artificial neural networks). In someimplementations, the points of observation 104 generate data streamsynthesized from the value that is changing over time. The data streammay also include pointers of references to programming language types orclosures that are used for modulating sensors and/or artificial neuralnetworks corresponding to the virtual being 110.

FIG. 1B illustrates a publish-subscribe model 130 of the process 100 ofsimulating sense data for virtual beings in virtual environments inaccordance with some implementations. In some implementations, thevirtual medium of observation 106 acts as a publisher by computing whatis sensed, based on location and types of the points of observation 104and sense properties 108. In some implementations, the sense properties108 are subscribers to a stream that informs the sense properties 108 todelete itself (from the list of sense properties of the virtual being110). In some implementations, the points of observation 104 can bedescribed as subscribers and the virtual medium of observation 106 canbe described as publishers. The virtual medium of observation 106produces (or generates) batch of data 132 that is input to one or morestreams. Each stream corresponds to a respective modulation (e.g.,modulations 134-2, 134-4, 134-6, . . . , 134-N). Each modulationcorresponds to a respective type descriptor (sometimes called sensorytype), that can be thought of as a channel name. For example, themodulation 134-2 corresponds to the type descriptor 142-2, themodulation 134-4 corresponds to the type descriptor 142-4, themodulation 134-6 corresponds to the type descriptor 142-6, and themodulation 134-N corresponds to the type descriptor 142-6. The streamsgenerate a batch of processed data 136 that is used to alter one or moreconfiguration parameters of the points of observation 10-4 and/or the AIand/or stream processing module 102.

In some implementations, programs that perform operations on the streamsare only subscribers, or they may be operators that are both subscribersand publishers, that take input, compute on or transform the input, thenpublish what is computed or transformed as their output. Operators mightmerge or split streams. Also AI programs perform operations on thestreams. In some implementations, AI programs (e.g., artificial neuralnetworks) are both subscribers and publishers in that the AI programsboth receive data or an effect in the stream, and may in turn publishback effect(s) to other streams. In some implementations, an effect isimplemented by way of a programmatic type, such as a class instance thatgets published to the stream, or alternately by an instruction insertedinto the stream to execute a programmatic procedure on the subscriberside. Some implementations use a framework called Combine (provided byApple's Swift language) that implements reactive programming andincludes notions of streams, and publishers and subscribers.

In some implementations, in the publisher/subscriber model, the senseproperties publish themselves to the relevant sensors when appropriateconditions are met (e.g., according to sensory type of the virtualmedium). For example, for heat or touch, the condition might beproximity; for visual, it might be line of sight. In someimplementations, alternatively or additionally, the sensors publishthemselves to relevant properties under certain conditions.

FIG. 2 is a block diagram illustrating a computing device 200 that canbe used to simulate sense data for virtual beings in virtualenvironments, according to some implementations. Computing devices 200include desktop computers, laptop computers, tablet computers, and othercomputing devices (e.g., smart phones, wearable devices), includingcomputer systems available on demand from a cloud computing service. Acomputing device 200 typically includes one or more processingunits/cores 202 (sometimes called processors; e.g., CPUs, GraphicalProcessing Units (GPUs), Application Specific Integrated Circuits(ASICs), System on Chips (SOCs), customized CPUs, such as customizedRISC-V CPU) for executing modules, programs, and/or instructions storedin the memory 210 and thereby performing processing operations; one ormore network or other communications interfaces 204; memory 210; and oneor more communication buses 206 for interconnecting these components.The communication buses 206 may include circuitry that interconnects andcontrols communications between system components, including any opticalcomponents. In some implementations, the computing device 200 includes adisplay 208 (sometimes called a display device) and/or one or more inputdevices 230. In some implementations, the input device includes akeyboard; in some implementations, the input device includes a “soft”keyboard, which is displayed as needed on the display 208, enabling auser to “press keys” that appear on the display 208. In someimplementations, the display 208 and/or input device 230 comprise atouch screen display (also called a touch sensitive display). In someimplementations, the display 208 is an integrated part of the computingdevice 200. In some implementations, the display is a separate displaydevice. Some implementations include cameras, microphones, and/ortactile devices, for interaction with human users and/or the real world.

In some implementations, the memory 210 includes high-speedrandom-access memory, such as DRAM, SRAM, DDR RAM or other random-accesssolid-state memory devices. In some implementations, the memory 210includes non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid-state storage devices. In some implementations, thememory 210 includes one or more storage devices remotely located fromthe CPUs 202. The memory 210, or alternately the non-volatile memorydevice(s) within the memory 210, comprises a non-transitory computerreadable storage medium. In some implementations, the memory 210, or thecomputer readable storage medium of the memory 210, stores the followingprograms, modules, and data structures, or a subset thereof:

-   -   an operating system 212, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a communication module 214, which is used for connecting the        computing device 200 to other computers and devices via the one        or more communication network interfaces 204 (wired or wireless)        and one or more communication networks, such as the Internet,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   one or more virtual being(s) 110 that includes a plurality of        sensory locations 216, a plurality of sense locations 218, and a        plurality of artificial neural networks and/or stream processing        modules 102. Each sensory location 104 includes one or more        sensors 104 (sometimes called points of observation). Each sense        location 218 includes one or more sense properties 108. In some        implementations, sense data streams (sometimes called stream        processing modules) include binary or text data (e.g., formatted        data, such as data formatted in JSON). In some implementations,        sense data streams are implemented using a protocol over a        medium, such as a computer network, computer bus, an optical        network. The sense data streams may stream data, or executable        binary code, or commands for performing operations, such as a        call or a remote procedure, or even source code that gets        interpreted or compiled and ran by the receiver. Typically, the        sense data streams include sensed data and effects (caused by        the sensed or sensory data) that are streamed from the sensors        to one or more artificial neural networks. In some        implementations, the sense data streams are included in a        program (e.g., a Swift program) which means that the sense data        streams encapsulate data and compiled class and structure and/or        enumerated data instances (or compositions thereof; sometimes        called compiled executable code). For example, a compiled class        instance that is streamed may carry out some effect on the        artificial neural network(s);    -   a virtual environment 220. Examples of the virtual environment        220 are described below in reference to FIGS. 3A-3D, according        to some implementations;    -   one or more objects and/or other virtual beings 112 that include        a plurality of sense locations 222, each sense location storing        one or more sense properties 224. Examples of the one or more        objects and/or other virtual beings 112 are described below in        reference to FIGS. 3A-3D, according to some implementations;    -   one or more virtual mediums 106 (sometimes called virtual medium        of observation). Examples of the one or more virtual mediums 106        are described above in reference to FIGS. 1A, 1B, and are        described below in reference to FIGS. 3A, 3B, 3C, 3D, 4A, 4B,        4C, 5, and 6 , according to some implementations;    -   one or more sensory types 142 (sometimes called type        descriptors). It is noted that each sensor and sense property of        the virtual being 110, each sense property of each object or        other virtual being 112, and each virtual medium has a        corresponding sensory type. Although FIG. 2 shows the sensory        types 142 as a separate data structure, in some implementations,        sensory type associated with each sensor, sense property, and/or        virtual medium may be a part of each data structure or module;        and    -   one or more modulations 134 that include one or more pain models        226 and/or one or more pleasure models 228. Examples of        modulations, pain and/or pleasure models are described below in        reference to FIG. 5 , according to some implementations.

Some implementations include a database 232 used by the computing device200 for simulating sense data for virtual beings in virtualenvironments. In some implementations, data sources of the database 232can be stored as spreadsheet files, CSV files, XML, files, flat files,JSON files, tables in a relational database, cloud databases, orstatistical databases.

Each of the above identified executable modules, applications, or set ofprocedures may be stored in one or more of the previously mentionedmemory devices, and corresponds to a set of instructions for performinga function described above. The above identified modules or programs(i.e., sets of instructions) need not be implemented as separatesoftware programs, procedures, or modules, and thus various subsets ofthese modules may be combined or otherwise re-arranged in variousimplementations. In some implementations, the memory 210 stores a subsetof the modules and data structures identified above. In someimplementations, the memory 210 stores additional modules or datastructures not described above.

Although FIG. 2 shows a computing device 200, FIG. 2 is intended more asfunctional description of the various features that may be presentrather than as a structural schematic of the implementations describedherein. In practice, and as recognized by those of ordinary skill in theart, items shown separately could be combined and some items could beseparated.

FIGS. 3A, 3B, 3C, and 3D show an example of simulating sense data forvirtual beings in virtual environments in accordance with someimplementations. FIG. 3A shows a virtual being 302 in a virtualenvironment 300 that includes two objects 304 and 306. The object 304 isspherical in shape, is deformable, has light or bright (brightness)sense properties (as indicated by the glow around its surface), and ismovable as indicated by the arrow pointing away. It is noted that thelight should not be confused with a 3D engine light source or anythingthat appears bright as rendered. The object may have an appearance ofbrightness, but the bright sense property is conveying a pain effect viaa bright virtual medium to sensors of sensry type bright, according tosome implementations. The object 306, on the other hand, is cuboidal inshape, is hot (as indicated by the waves propagating away from the topof its surface), and is static (or not movable). It is noted that anobject that has an appearance when rendered by a virtual camera of beinghot is not the same thing as an object with a temperature sensory typein a high range, according to some implementations. In FIG. 3A, thevirtual being (e.g., a model of a human; sometimes called an avatar) hasits head turned towards the object 304. FIG. 3B shows a view when theobject 304 is about to land on a palm of the virtual being 302, FIG. 3Cshows the object 304 landing on the palm of the virtual being 302, andFIG. 3D shows the object 304 deforming (sometimes called soft bodydeformation) based on an impact between a surface of the object 304 andthe palm. It is noted that although FIGS. 3A and 3C do not show eyes,nose, mouth, tongue, teeth, eyebrows, hair, fingernails, and many otherphysical body feature details, such model details will be used in someimplementations for simulation of the virtual being 302. According tosome implementations, body regions of the virtual being 302 have senseproperties affecting points of observation on itself and other bodies orobjects, and include one or more type descriptors (e.g., temperature,light touch, pressure, stretch). The virtual being 302 includes severalpoints of observation, each associated with type descriptor (e.g.,temperature, light touch, pressure, stretch/compress). In someimplementations, the three-dimensional space (e.g., region 308 shown inFIG. 3B) between the object 304 and the virtual being 302 can beassociated with one or more virtual medium of observation with distincttype descriptors (e.g., temperature, light touch, pressure, vibration),used to simulate distance and/or soft-body deformation (oscillatorysoft-body deformation for vibration) when surfaces come in contact. Theobject 304 includes multiple sense properties (e.g., sense propertieswith type descriptors temperature, light touch, pressure, vibration, andstretch).

Some implementations use a model for an avatar. Some implementations useskinning, skinning deformers, skin cluster, and/or skeletons (e.g.,skeleton made using Autodesk Maya or similar 3D computer animationsoftware with modeling, rendering, simulation, texturing, and animationtools used by artists, modelers, or animators). Some implementations useone or more rigs made using an iterative rigging framework, such asmGear. In some implementations, the model, skinning and mesh areexported from a 3D computer animation software, such as Maya, andimported into a high-level 3D graphics framework (e.g., Apple SceneKit)used to create 3D animated scenes and effects in mobile applications.Some implementations use SceneKit as the virtual environment for aprototype, and SceneKit can be used for both iOS and Mac OS programs.Some implementatiions use a Mac OS Swift program or Multiplicity thatuses the SceneKit framework for the simulation engine providing gameengine features. Some implementations use Unreal Engine. Someimplementations use Unreal Engine and SceneKit Mac OS apps. In someimplementations, the rig (e.g., rig with solvers) is also recreated inthe 3D graphic framework (e.g., SceneKit) to programmatically driveanimation. In some implementations, portions of the iterative riggingframework are ported to or implemented in the 3D graphics framework. Insome implementations, the rig is driven by one or more machine learningalgorithms, systems, and/or programmed algorithms (e.g., the AI brain).In some implementations, Swift-based APIs for the avatar and AI brainInterface are in the SceneKit context. Some implementations use one ormore SceneKit classes with the avatar, such as SCNIKconstraint,SCNSkinner, SCNMorpher, SCNShadable, SCNProgram, and/or classes thatprovide similar functionalities.

In various implementations, the virtual being 302 is a rigged avatar ofa human body, a virtual approximation of a body of human, an animal, aninsect, a humanoid, or a creature, an object with virtual mobility, anobject that is dynamically altered in shape, size, or form, an objectwith a skeleton and skinning, an object that is altered using skeletonjoints, blend shapes, rigging and its articulation including solvers,such as Forward Kinematics (FK) solver and Inverse Kinematics (IK)solver, and/or programs that trigger baked animations that update jointvalues during animation, and various soft body deformation. In someimplementations, the virtual being 302 may employ ArtificialIntelligence (AI) or Machine Learning (ML) algorithms to learn to moveitself in the virtual environment 300 by actuating and/or modulatingvalues for the rig/rigging system, dynamically triggering bakedanimation that change joint values, and/or dynamically actuatingmovement or using action control APIs present in the virtual environment300.

In some implementations, when simulating sense data using a virtual body(sometimes called a virtual being or an avatar) is compute intensive,some of the simulation is offloaded to a real-time 3D creation platform(e.g., Unreal Engine) and/or a computation platform that enables thecreation of massive simulations and virtual worlds for use in videogames (e.g., Improbable's SpatialOS), or a similar cloud game engineplatform. Some implementations use a bridge to extend SpatialOS typecapabilities to SceneKit.

In some implementations, the virtual body 302 has eyes, ears, nose,mouth and skin senses that approximate the human eyes, ears, nose, mouthand skin senses, in form and/or function. In some implementations, amajority of systems of the human body have associated approximationmodels in the virtual body and/or the AI brain.

Some implementations use stock 3D models used in 3D graphics, such asmodels supplied by TurboSquid, and/or augment the models with one ormore sense properties.

Some implementations build a human body approximation using 3D gameengine and computer animation and modeling technology that provide AIbrain algorithms with sense data. The virtual body is controlled by theAI brain algorithms. Some implementations use the AI brain algorithms tocontrol the skeleton, skinning, mesh, and/or body rigs, using InverseKinematics solvers (IK solvers) and similar solvers. In someimplementations, sense systems of the body and the AI brain algorithmsinter-modulate. Some implementations simulate details, such as breathingand inter-modulation of various bodily systems. Some implementations useprebaked animations, and a rig and/or rigging system that runs in thecontext of the game engine. Typically, the rig and/or rigging systemsare used in an animation and modeling tool by an animator. In someimplementations, for this approximation, the rigs are programmaticallyactuated using machine learning algorithms.

In some implementations, the virtual environment 300 includes surfaces,manifolds, and/or point clouds. In some implementations, the virtualenvironment 300 is simulated using a game engine, a simulation engine,physics and/or rendering engines. In some implementations, the virtualenvironment 300 is a rigged environment. In some implementations, thevirtual environment 300 is simulated using a cloud platform game andsimulation engine, where simulation and rendering are partitioned andhandled by cloud servers and cloud infrastructure.

In some implementations, the virtual environment 300 is created for theavatar (sometimes called the virtual body or the virtual being 302) tointeract with. In some implementations, the virtual environment 300 isreal-world like, and includes simulations and/or approximations ofreal-world things. For example, some implementations simulate air and/orwater. Some implementations use real fluid and/or air flow simulations.Some implementations provide tools for adding real-world properties tomodels. In some implementations, such properties comprise weight,temperature, taste, texture, and odor. Some implementations definesensors, virtual mediums, and/or sense properties for taste sensorytypes (e.g., salty, sour, sweet, umami, astringent, pungent, bitter),and/or for olfactory or smell sensory types (e.g., sweet, fragrant,woody/resinous, fruity (non-citrus), chemical, minty/peppermint,popcorn, lemon, decaying (sickening), pungent (sickening)).

In some implementations, objects that come into contact with otherobjects create sounds and the sound propagation is approximated withsound ray tracing and/or sound path tracing techniques.

Some implementations extend 3D polygon mesh assets with real worldproperties that can be sensed. Examples of such properties includeweight, temperature, taste, texture, and odor. Some implementationssimulate sound. It is noted that, in some implementations, althoughweight is already a game engine gravity effect, the weight is alsosensed by the virtual sensors with sensory types pressure and lighttouch.

Some implementations simulate common real-world things or objects thatare solid, liquid, or gas. In some implementations, such things orobjects exhibit sense data when contacted by and/or interacted with thebody. In some implementations, contact is by way of a simulated mediumof observation approximating the real world mediums of observation likelight, sound, or certain types of molecules that can be sensed by asimulated olfactory and taste system.

Some implementations extend conventional techniques that simulate sensedata using mathematical models of human body parts, such asthree-dimensional (3D) body finite-element models that use discretepoint data generated using body scanning. Some implementations simulatetactile signals for a hand (e.g., with millisecond precision), and use3D finite-element models of humans and/or animals. For example, someimplementations use models of monkey fingertips to simulate themechanics of tactile sense. Some implementations use data from smartskin and/or nanowire sensors.

Some implementations model real-world objects using fundamentalqualities. Such objects are made of atoms or molecules. The fundamentalqualities of objects can be observed by point(s) of observation by wayof medium(s) of observations. In some implementations, the mediums ofobservation (sometimes called virtual mediums of observations) arevirtual substitute mediums for real world mediums of light, sound, andforces.

Some implementations use machine learning algorithms that are“fundamental quality representation and recreation algorithms” (FQRR)that accept data input, learn to predict, classify, store encodedpatterns, then learn to recreate the patterns without external input,are algorithms that can be used for strong AI, as opposed to machinelearning algorithms that are most often “label representationalgorithms” that accept data input, and only learn to predict andclassify patterns.

In some implementations, fundamental qualities are intrinsic to anobject in that they exist independent of how they are being observed.Fundamental qualities include the qualities of action reaction betweenobjects. Any particular wave is also a fundamental quality. Wave isdefined as a disturbance moving through a medium.

In the human body, by way of points of observation in the body, thebrain is observing, by way of a medium of observation, such as light, aslice of what exists in reality and is attempting to recreate the“information” that are the fundamental qualities of the thing that isobserved. A fundamental quality can be that an object has a certainshape with specific salient features, or that it absorbs certainwavelengths of light while reflecting other wavelengths of light whichhere is our medium of observation. Salient features of an object arealso fundamental and emergent qualities. Salient features are differentat different scales. At the atomic scale, they are things likewavelengths being absorbed, or reflected or re-emitted. Of course, ifthe medium of observation is only reflected light, then not all of thefundamental qualities are observed and recreated.

To further illustrate the concepts, suppose a human observes a metalsign pole and walks over to it and tastes it, then the human isobserving another fundamental quality. When the human tastes something,the human observes some fundamental qualities through the taste pointsof observation (taste buds) and transforms it into a perception.Perceptions are not merely a label (e.g., the thing observed is sweetand has X quantity), but rather the micro scale shape of the moleculesin food and their energy states as sampled, encoded and recreated by thehuman brains, processes that are referred to as fundamental qualityrepresentation and recreation (FQRR) algorithms. One key differencebetween a brain and a computer is that the brain captures and recreatesthe essence of objects in the real world with its form of FQRRalgorithms, whereas computer recognition programs and most weak AI arelabel representation (LR) algorithms. The term essence in this contextmeans a representation of some fundamental qualities that reflectsaccurately or is at least an approximation of what the form of the thingobserved, versus just a label. Some implementations use FQRR algorithmsthat use observed fundamental quality information propagated by way of amedium of observation and as observed by some number of points ofobservation in a body. In some implementations, a FQRR algorithmrecreates the fundamental qualities. In some implementations, a FQRRalgorithm recreates a partial and/or incomplete slice of the realityobserved by the human (or the virtual body). FQRR algorithms are centralto perception, and active perception leads to awareness.

In nature, objects absorb certain wavelengths of light and reflect thelight having wavelengths that are not absorbed. The medium ofobservation, the light, reaches the lens of our eyes and is focused onthe retina and fovea. The eye is mechanical in that it is made toextract the fundamental qualities of the object being observed, such ascolor, shape, and salient features. However, the object has otherfundamental qualities that are not being observed. For example, theobject may provide sense data by way of the medium of observation,points of observation (for the type of sense data). In someimplementations, a FQRR algorithm processes this sense data to obtain ataste and/or smell perception.

Compounds that are senses with taste olfactory senses also exist inreality. Such compounds are comprised of atoms and molecules. When tastebuds and olfactory senses come into contact with a substance, they areextracting and conveying some set of fundamental qualities about thesubstance. That is its shape, and salient features that are fundamental.The brain then takes this information about the fundamental qualitiesand attempts to reconstruct those fundamental qualities from andas-stored representations. Some implementations mimic this humanbehavior using memory that is multidimensional (e.g. temporal andspatial), and/or using recursive computations. In some implementations,a result of the computation(s) is a literal recreation of thefundamental qualities as observed (salient features as observed throughpoints of observation).

To facilitate simulation of an approximate body, some implementation usethe following definition of observation and points of observation. Inthe human body, nerve cells are points of observation. Observation iscomplex and involves body topology, makeup, and the topology and kind ofthe points of observation.

It is noted that the FQRR algorithms described above are not a mererecording of information. The FQRR algorithms involve the type of pointsof observation, and their topology in the body. Further, the FQRRalgorithms include encoding and storing what is observed, and decodingand re-creation of information from a representation or set ofrepresentations previously encoded. Algorithms that involve discreteencodings without the mechanisms to decode and assemble them intosomething greater are LR algorithms. LR algorithms are pruned and alimited number are encoded and stored in what is roughly analogous todictionaries. FQRR algorithms use dynamic dictionaries of LR algorithms.

In some implementations, points of observation are points on a body thatobserve. For example, the retina and fovea are a surface with points ofobservation. In some implementations, the tongue is another topologyhaving taste buds of different kinds as points of observation.

Points of observation observe fundamental qualities of reality by way ofa medium of observation. For example, the human eye retina and fovea area surface containing points of observation that are rods and cones. Byfocusing the medium of observation, light, on the retina, the eye lenscaptures the fundamental qualities of something in reality, like shapeand color of objects. Some implementations calculate relative locationin a 3D space as a dynamic quality, processed by a plurality of FQRRalgorithms. Some implementations use a movable virtual body.

To further illustrate FQRR algorithms, consider an image of a collage ofphotos that resembles the iCub robot (e.g., the well-known iCub image).Each photo in the set or collage can be considered to be an LRalgorithm, and the set can be considered to be a dictionary. Supposefurther that the set only allows one of each image to be stored. Theentire collage can also be considered an LR if stored as an array ofpixels. A FQRR algorithm observes the iCub robot in reality, then by wayof using previously encoded and stored images in the set, the algorithmcan recreate the representation of the collage. The FQRR algorithm canalso identify the collage when presented with internally generated datasignals that create an image of the iCub robot.

To illustrate that active perception is awareness to some degree,suppose a person's hand touches something hot, sense data from thenerves of the hand allows the person to form a perception of hotness.The response is to pull the hand away. The prediction that pulling thehand away will take away the perception of hotness and replace it withanother anticipated perception is active perception or awareness. Toillustrate further, consider another example. When a person turns theirhead to see someone in the room they previously saw, they have formed apartial perception. When they see the person, the perception is fullymanifested, and this entire process is active perception and awareness.Otherwise, they would just turn their head and be surprised. Perceptioninvolves FQRRs which involves memory and the ability to recreate.Reaction alone is not awareness as it involves no perception only ameasurement and an instruction. For example, a robot arm with a heatsensor may have an instruction to retract the arm when the temperatureread by the sensor is above 200 degrees C.

Qualia are individual instances of subjective, conscious experience,such as pain due to a headache, or the taste of a wine. Perception asdescribed herein is qualia. A wavelength of light that is called reddoes not have red. What red is, is a higher level perception of how theobject is interacting with the medium of observation. Red is qualia asis everything that the brain interprets with its form of FQRR. Lightwaves in a range and frequency labelled as red is not the red qualia orred perception. Perception of red is an emergent property of FQRR.

Mental model corresponds to perceptions that in turn correspond toqualia. It does not objectively matter that green is seen as green or asred, when referring to a thing in reality that absorbs blue and redlight and reflects green light because that thing exhibits certainabsorption and reflection as a fundamental property. Someimplementations build a mental model, a FQRR, of observed sense data,and that mental model is created the same way for the human species, sothe mental model is equivalent to perception which is in turn equivalentto qualia. In other words, the mental model that is equivalent toperception which is in turn equivalent to qualia of red is the same fordifferent individuals, because the architecture for sensing (the pointsof observation in the body) and creating the mental model is the same.Another species, such as a butterfly, likely does not have the samemental model and perception of red as humans do. Butterflies and beessee colors that humans do not see. Qualia that is equivalent toperception that is equivalent to mental model of any color is likely tobe different across a species. Humans form no mental model of colorsthat they don't sense (i.e., have points of observation for).

Common frames of reference is having mental models that are equivalentto perceptions that are equivalent to qualia that are similar but notnecessarily equivalent. For example, as described above, humans do notsee all the colors that butterflies see and therefore do not have theperceptions of these colors. However, humans do have perceptions ofother colors so humans form higher level perceptions that canextrapolate to a degree what it means to have a perception of anothercolor that they don't see.

Common frames of reference is equivalent to similar mental model whichis equivalent to similar perception which is in turn equivalent tosimilar qualia. Pain is perception and although pain perception maydiffer between humans, they are similar enough to say that humans have acommon frame of reference. Common frames of reference is inclusive ofthe cases where perception of something is likely to be nearly exactsuch as when two humans are perceiving colors, versus only similar as itwould be between a butterfly and a human. That is, common frames ofreference is inclusive of the exact model which is equivalent to exactperception which is in turn equivalent to exact qualia and similarmodel, similar perception, and similar qualia.

Common frames of reference facilitate quicker transfer of mental models,perceptions, and qualia between humans or an AI. Without common framesof reference, language does not work well in conveying or transferringperception. Common frames of reference, perceptions of languageconstructs associated with other perceptions, make it easier for thebrain to internally recreate sense data that gets turned back intoperception. Common frames of reference is applicable to having similarsets of perceptions that contain similar perceptions.

With regards mental models as it relates to scale, the mental models ofsound and vision differ partly because they are at different scales. Asa medium of observation, sound travels slower and is more local in itsinteractions with things in reality. Humans' perception of sound isdifferent than a bat's perception, because the bat is using sound tonavigate. A bat's perception of sound is visual, whereas humanperception of sound is not visual. In other words, a bat's brain FQRRsare creating a slice of reality that recreated the spatial relationshipsof things around it. The set of salient features that are pertinent tonavigation are different than the salient features pertinent todifferentiating sounds. Sometimes called “computed salient features”,these are more or less higher-level perceptions and or artifacts oftopology and the type of the points of observation.

Some implementations take into account topology in the body and type ofthe points of observation is what provides the sense data needed to formperception. To form perceptions that require certain sense data, suchsense data needs to be acquired. If the topology and type of points ofobservation between two systems corresponding to two different speciesare different, the processing of information by their body's points ofobservation differ in that they don't extract similar salient features,then the mental model or perception or qualia will be different.

There are emergent salient features in reality when viewed at differentscales. For example, a sign on the road is made up of trillions ofatoms. At the atomic scale, there is no sign but just atoms. Taken intotality at a different scale, there are the emergent features of thesign and the shapes. The difference between seeing in black and whiteand seeing in color is that in one case there is a mental model,perception, or qualia based on sensory data from points of observationthat are of a type that is equally sensitive to all visible wavelengths.Whereas, with color sensory data, there are three different kinds ofpoints of observation, each sensitive to red, green or blue wavelengths.The mental model, perception, or qualia differentiates the colorspectrum. Put another way, perception is built from sense data offundamental qualities of spatial separation between points or thingsbeing observed in reality, and the light absorbing and reflectingfundamental qualities of those things or points.

Awareness of something arises through active observation and perception.In some implementations, the set of FQRRs has mechanisms that directactive observation and predict perception, thereby generating or causingawareness.

Some implementations include AI-based language capabilities (e.g.,natural language processing skills). Some implementations master suchskills by acquiring an approximation of the mental model, perception, orqualia of experience. The quality of any AI's language capabilities iscorrelated with the richness of its experience spectrum. As describedabove in the Background section, to have a rich set of experiencesacross a spectrum requires the AI to have a body with a large set ofsenses and an environment that closely approximates the real world or isthe real world. Since inserting the AI into a mammal body is notpossible at the current time, some implementations use a virtual bodyand environment.

Some implementations use pain and/or pleasure models as primary mentalmodel, perception, or qualia. In some implementations, pain and pleasureperceptions results from complex sense data (sensation) from the bodywhich includes the AI brain. In some implementations, pain perceptionsare generated from sense data that correlates to be generally bad forthe avatar body and the AI brain. In some implementations, pleasureperceptions are generated from sense data that correlates with theavatar body and brain being in a state of well-being.

In some implementations, sense data has emergent salient features thatare directly related to the topology of the body and its points ofobservation. In some implementations, further active control of the bodyor parts of the body creates new topology and dynamic salient featuresand perceptions. To illustrate, when a person blows up their mouth withair, that is an active control that makes for stretching, that causesnerves in the face to send sense data to the brain. It is differentthan, for example, when light impacts the cones and rods of the retina.In the case of the retina, the sense data is passive since it is fromobservation of things external to the body and it is not activelychanging the topology and firing of the nerves.

FIGS. 4A, 4B, and 4C show an example of simulating human vision(sometimes called eye simulation) according to some implementations. Forthe sake of illustration, suppose the virtual medium of observation is arendered frame output from a camera. The output affects points ofobservation mapped to the rendered output. In the description thatfollows, the virtual medium of observation is rendered frames, whereaswith skin senses (light touch, pressure, vibration, heat, stretch), thevirtual medium of observation computes soft body deformation. In thisway, the virtual medium of observation can differ depending on what isbeing virtually sensed. Some implementations use points of observation,virtual medium of observation, and sense properties to simulate humaneye retina and/or fovea, according to techniques described herein.

The human retina contains about 120 million rod cells, and 6 millioncone cells. Some implementations approximate the human eye using avirtual camera similar to how game and animation engines use the conceptof a virtual camera and virtual lighting sources. FIG. 4A shows anexample 400 of a camera frustum 402 (sometimes called a viewingfrustum), according to some implementations. A virtual camera 404includes a Field of View (FOV) is pointed at a three-dimensional (3D)space as indicated by the three axes, x-axis (406), y-axis (408), andz-axis (410). FIG. 4A also shows angles for the field of view (e.g., xFOV 412, and y FOV 414), two planes of view, z Near plane 416 (closer tothe camera) and z Far plane 418 (further away from the camera along thez-axis). FIG. 4B shows a virtual lighting source 420, according to someimplementations. In FIG. 4B, the virtual lighting source 420 is shownaligned with the y-axis 408, in some implementations.

In some implementations, to simulate a 3D scene (sometimes called avirtual environment), virtual cameras are associated with fixedlocations or attached to objects that move or otherwise moved about. Insome implementations, objects in the scene that fall between the z Nearplane 416 and z Far plane 418 of a camera are rendered out to frames,typically at 60 frames per second. In some implementations, objects orsurfaces (or part thereof) in a 3D scene are associated with senseproperties (for example, with type descriptor bright).

In some implementations, when objects are between the z Near plane 416and z Far plane 418, the objects get rendered, and if they haveadditional sense properties of sensory type bright, then affect pointsof observation with sensory type bright. The additional sense propertiesmight correspond to the X and Y pixel locations of each rendered frame.Alternately, in some implementations, points of observation are mappedto a region of pixels for each rendered frame. For example, each pointof observation is mapped to 2-by-2 pixels or 3-by-3 pixels, instead of aone-to-one mapping.

In some implementations, for eye and/or retina simulation, pixel valuesof each frame that are part of a stream of rendered frames are postprocessed and used to set values for associated points of observation tocause the points of observation to have light responses andcharacteristics that approximate the response characteristics of rodsand cones in the human eye. In some implementations, as discussed above,additionally, those points of observation may also have one or more typedescriptors that match with the type descriptors of sense propertiesassociated with objects in the scene currently being rendered. Forexample, the points of observation mapped to an (X, Y) pixel coordinatesof rendered frames may have a type descriptor bright. If an object beingrendered also has sense properties of type descriptor bright, then theywould have an additional affect (modulation or attenuation) on thepoints of observation, and potentially a modulating or attenuatingeffect upstream of the points of observation on programs that performoperations on the stream, and/or AI programs (e.g., artificialintelligence programs, sense data stream processing, artificial neuralnetworks and/or stream processing modules 102).

FIG. 4C shows the virtual camera 404 pointed at a cube 428, according tosome implementations. Position of the cube 428 is indicated by a line+X(430) to −X (432) aligned with the x-axis 406 (shown in FIG. 4A), line+Y(434) to −Y (436) aligned with y-axis 408 (shown in FIG. 4A), and line+Z(438) to −Z (440) aligned with the z-axis 410 (shown in FIG. 4A). Insome instances, the cube 428 is a normal cube in which case it isrendered into each frame, and then that stream of frames would affectmapped points of observation to simulate response and/or characteristicsof human retina rods and cones. In other instances, a similar cube caninclude sense properties (e.g., type descriptor bright) mapped onto oneor more of its six surfaces. If the points of observation mapped to therendered frame output of the camera also have the same type descriptor(i.e. bright), then the points of observation are affected in someadditional way, via attenuation or modulation, and/or according to oneor more pain and/or pleasure models, as described below in reference toFIG. 5 , according to some implementations.

FIG. 5 illustrates example modulation effects 500, in accordance withsome implementations. Some implementations use a pain and/or pleasuremodel for amplifying, attenuating, and/or suppressing body relatedperceptions. To illustrate, suppose a human (or the virtual body) ispricked by a pin. The pain prick will cause pain. In this case, the painstimulus is localized (to the region of the pin prick). Pain thereforecan be caused by a small number of nerves in a very localized part ofthe body. Pain suppresses or attenuates other body related perceptions,typically suppressing sense data from or along main nerves or possiblyin the brain. In simple organisms, the pain may only suppress othernearby nerves. Suppressing sense data suppresses perceptions, whileamplifying the pain perception at the same time. On the other hand,pleasure amplifies, expands, increases body related perceptions, turnsup perceptive awareness related to the body. Some implementations modelpleasure caused by seeing an object or a characteristic of an object.For example, some implementations model a visual effect or pleasurecaused by seeing a color (e.g., green) in a scene (i.e., the virtualenvironment).

In some implementations, the modulations 134 include modulating programs502 (e.g., closures, pointers, references to program language types orcompositions of types that are instantiated and executed) that applymodulation effects 504 at target of effects 506. Example target ofeffects 506 include the AI and/or sense data stream processing 102, andthe points of observation 104, according to some implementations. The AIand/or sense data stream processing 102 are associated with types ofmodulations associated with a pain model 508, and/or types ofmodulations associated with a pleasure model 510. Similarly, the pointsof observation 104 are associated with types of modulations associatedwith a pain model 512, and/or types of modulations associated with apleasure model 514, according to some implementations. The points ofobservation 104 are also associated with regions of modulations (e.g.,points of observation corresponding to the virtual being 110, objectsand other virtual being 112, and non-objects 114).

Example Pain and Pleasure Models

This Section describes example pain and pleasure models, according tosome implementations. In the description that follows, example times arespecified using real-time seconds (or wall clock time). If thesimulation is run slower or faster than real-time, then the exampletimes need to be adjusted accordingly.

In the description that follows, size of the virtual being, ratios ofbody part sizes, and dimensions generally correspond to real world beingequivalents. Parameters specified are assumed to be tunable for a givensimulation. Sensory type is assumed to be a label that is descriptive ofthe virtual medium interaction. For the sake of illustration,deformation assumes a movement of sensory and sensory locations in adirection normal to the baseline location. It is further assumed thatthe baseline location is a manifold or shape.

In some implementations, pleasure is a reward that has one or morecauses and one or more effects. An example cause for pleasure is, inaccordance with interactions between sensors in sensory locations on avirtual being, and sense properties in sense locations on the samevirtual being or another virtual being or object, where sensory type ofboth sensors and sense properties is light touch, where if the sensorylocations and sense locations are on the same virtual being then they donot interact when they are in the same location or overlap. If thesensory locations and sense locations are not on the same virtual being,then interaction will occur even if they are the same location oroverlap. A deformation to the sensory locations is in a range of 1nanometer to 5 millimeters in a direction normal to the sensorylocation's non deformed baseline location within time periods of 1/50thof a second to 1 second, activates a corresponding pleasure effect. Insome implementations, the corresponding pleasure effect is to attenuateup by 1 percent to 200 percent, as determined by the artificial neuralnetworks, resolution of the sensors in the current set of pleasure causeactivated sensory locations, and/or to attenuate up by 1 percent to 200percent the resolution of the sensors in other non-pleasure causeactivated sensory locations.

Another example cause for pleasure is, in accordance with interactionsbetween sensors in sensory locations on a virtual being, and senseproperties in sense locations on the same virtual being or anothervirtual being or object, where sensory type of both sensors and senseproperties is temperature, if the sensory locations and sense locationsare on the same virtual being, then they do not interact when they arein the same location or overlap. If the sensory locations and senselocations are not on the same virtual being, then interaction will occureven if they are the same location or overlap. In some implementations,when distance between sensory locations and sense locations is less than1 foot, then the interactions may activate a pleasure effect accordingto the following table:

Deformation normal to the Tempera- sensory ture location's non rangeDistance deformed Temperature's pleasure effect  0-130 Less than 0.0 mmNone 1 ft. 90-100 Less than 0.0 mm For each degree in the range 1 ft.above 90, attenuate up by 0.05 percent the resolution of the sensors inthe current set of pleasure cause activated sensory locations 90-100Less than 1.0 nm-5.0 mm For each degree in the range 1 ft. above 90,attenuate up by (20.0 percent times deformation/5.0 mm) the resolutionof the sensors in the current set of pleasure cause activated sensorylocations

Another example cause for pleasure is, in accordance with interactionsbetween sensors in sensory locations on a virtual being and senseproperties in sense locations on the same virtual being or anothervirtual being or object, where sensory type of both sensors and senseproperties is vibration, where if the sensory locations and senselocations are on the same virtual being, then they do not interact whenthey are in the same location or overlap. If the sensory locations andsense locations are not on the same virtual being, then interaction willoccur even if they are the same location or overlap, where thecalculated vibration frequency of the manifold region of the sensorylocations more than 0.5 Hertz (Hz), activates a corresponding pleasureeffect. In some implementations, the corresponding vibration pleasureeffect is to attenuate up by a percentage time the resolution of all thesensors of all sensory type in the virtual being's vibration affectedsensory locations to a maximum of 10 percent in 10 minutes after whichthere is no further pleasure effect for 1 hour, according to theirrespective calculated vibration frequency of the manifold region,according to a formula x=f{circumflex over ( )}1.1+1 for every 10seconds the vibration presents, where f=frequency.

In contrast to pleasure, pain is an anti-reward that has causes andcorresponding effects, examples of which are provided below forillustration, according to some implementations. An example cause forpain is when a pleasure activation of (e.g., activation of sensory typelight touch) does not occur within a virtual being within a time period(e.g., 30 seconds), then the pleasure effect activates a pain effect. Anexample pain effect is to attenuate down by 1 percent the resolution ofall the sensors in the virtual being where sensory type is light touch,according to some implementations.

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations on a virtual being and senseproperties in sense locations on the same virtual being or anothervirtual being or object, where sensory type of both sensors and senseproperties is pressure, where if the sensory locations and senselocations are on the same virtual being, then they do not interact whenthey are in the same location or overlap. On the other hand, if thesensory locations and sense locations are not on the same virtual being,then interaction occur even if they are the same location or overlap.Similarly, when deformation to the sensory locations is in a range ofdistance (e.g., 6 millimeters or greater) in a direction normal to thesensory location's non deformed baseline location within predeterminedtime periods (e.g., 1/60th of a second up to 10 seconds), interactionactivates a corresponding pain effect. For example, the pain effect isto attenuate down by 4 percent for every 1 millimeter of deformationbetween 6 millimeters and 16 millimeters, and 6 percent for every 1millimeter of deformation between 16 and 26 millimeters, the resolutionof the sensors in the virtual being with sensory type light touch andpressure that are within the pain activated sensory locations, forperiod of 5 minutes. In some implementations, if the resolution of anyof the sensors ever becomes 0, then they are removed from the virtualbeing.

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations on a virtual being and senseproperties in sense locations on the same virtual being or anothervirtual being or object, where sensory type of both sensors and senseproperties is temperature, where if the sensory locations and senselocations are on the same virtual being, then they do not interact whenthey are in the same location or overlap. On the other hand, if thesensory locations and sense locations are not on the same virtual being,then interactions occur even if they are the same location or overlap,and the interactions may activate a corresponding pain effect accordingto the following table:

Deformation normal to the sensory location's Temp- non erature deformedrange Distance baseline Temperature's pain effect. 50-130 Less than 0.0mm None 131 and Less than 0.0 mm For each degree in the range above or 1ft. above 131 or below 50, below 50. attenuate down by 0.33 percent theresolution of the sensors in the current set of pain cause activatedsensory locations 131 and Less than 1.0 nm- For each degree in the rangeabove or 1 ft. 5.0 mm above 131 or below 50, below 50. attenuate down by(1.0 percent * deformation/5.0 mm) the resolution of the sensors in thecurrent set of pain cause activated sensory locations

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations on a virtual being and senseproperties in sense locations on the same virtual being and samemanifold, where sensory type of both sensors and sense properties isstretch or compress. In order to interact, the sensory locations andsense locations must be part of the same virtual being and not have thesame baseline location or have overlapping baseline locations. When thesensory locations and sense locations' respective baseline locationclosest edge points lie within a predetermined distance (e.g., 2 mm)apart, along a closest path between them on a defined manifold whicheach are on, then the interactions may activate a corresponding paineffect according to the following table:

Sensory location's and sense location's baseline location closest edgepoints distance apart (as calculated along a closest path between themon a defined manifold which each are on) (sensory type = stretch orcompress, manifold is typically the outer manifold of the virtual being)Pain effect 2.0 mm or greater apart. None Less than 2.0 mm apart. Foreach 0.05 mm increase or decrease from the baseline closest pathdistance, attenuate down by 1 percent the resolution of the sensors ofall sense types in the current set of stretch or compress pain causeactivated sensory locations.

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations on a virtual being and senseproperties in sense locations on the same virtual being or anothervirtual being or object, where sensory type of both sensors and senseproperties is vibration, where if the sensory locations and senselocations are on the same virtual being, then they do not interact whenthey are in the same location or overlap. If the sensory locations andsense locations are not on the same virtual being, then interaction willoccur even if they are the same location or overlap if the calculatedvibration frequency of the manifold region of the sensory locations ismore than a predetermined frequency (e.g., 0.5 Hz), and activates acorresponding pain effect. In some implementations, the correspondingpain effect is to attenuate down by a percentage times the resolution ofall the sensors of all sensory types in the virtual being'svibration-affected sensory locations according to their respectivecalculated vibration frequency of the manifold region, according to aformula, where x=f{circumflex over ( )}1.1 every 10 seconds thevibration presents, where f=frequency.

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations representing the retinas and retinasubstructures of the virtual being's eyes and sense properties in senselocations on the same virtual being or another virtual being or object,where sensory type of both sensors and sense properties is bright, wheresensors are receiving input from one or more respective virtual mediums,where the virtual mediums are virtual cameras that move and change theirparameters in accordance with the virtual being's virtualized eyemovements, virtualized eye focusing, virtualized eye dilation, and theinput to the sensors is the respective rendered frame pixel output fromthe viewing frustums of the respective virtual cameras, either in rawdata form or processed first through artificial neural networks, andwhere input to sensors only occurs when the sense properties in senselocations of sensory type bright are within the viewing frustums of thevirtual cameras, this activates a corresponding pain effect according tothe following table:

Duration activated sensors continuously receive bright input Bright paineffect 0.0-1.0 second None 1.01-30.0 Temporarily attenuate down by 2percent per second the resolution of the sensors of sensory type“bright” in the activated sensory locations. If the activated sensorylocations become inactive attenuate up by 2 percent per second, thepreviously attenuated down sensors until they again are at theirmaximum. The maximum can be less than 100 percent of their originalresolution if the maximum has been lowered. Above 30.0 Temporarilyattenuate down by 2 percent per second seconds the resolution of thesensors of sensory type “bright” in the activated sensory locations andpermanently reduce the maximum resolution by 1 percent per second. Ifthe activated sensory locations become inactive attenuate up by 2percent per second, the previously attenuated down sensors until theyagain are at their maximum. The maximum can be less than 100 percent oftheir original resolution if the maximum has been lowered.

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations representing the virtual ear drum,the attached virtualized malleus, incus, stapes (ossicles), cochlea andcochlea substructures of the virtual being's ears and sense propertiesin sense locations on the same virtual being or another virtual being orobject, where sensory type of both sensors and sense properties issound, where sense locations with sensory type sound emit sound that isray traced or path traced in the virtual environment, where sensors arereceiving input from one or more respective virtual mediums, where thevirtual mediums are sound ray tracing or path traced audio from thesense locations with sensory type sound arriving at the virtual being'svirtualized ear drum, where the arriving audio is accordingly modeled tovibrate the eardrum, where the vibration of the eardrum is accordinglymodeled to move a virtualized malleus, incus, stapes or substitute thatcombines all three, where the stapes or substitute sits against a secondvirtual inner membrane that sits between it and a virtualized fluid inthe cochlea, where the inner membrane in contact with the virtual fluidis accordingly modeled to disturb the fluid, where the virtualized fluidis near to or in contact with thousands of virtualized tiny hairs alsoin the virtual cochlea, where the contact and interaction of the virtualfluid on the hairs is accordingly modeled using a large number of soundsamples covering a large range of frequencies and amplitudes and types,where a set of virtual cameras have the hairs within their viewingfrustums and capture and render out frame data, where the frame data isfed into a first set of artificial neural networks training them torecognize the features of individual hairs and classify their respectivemovements in the fluid and tensions according to specific soundfrequencies and amplitudes and sound types affecting the fluid, where asecond set of artificial neural networks is used to modify and optimizethe topology of the virtualized ear components for optimized receivingof the ray or path traced audio, where a third set of artificial neuralnetworks is used to modify and optimize the shape, placement, size anddistribution of hairs in the virtual cochlea to optimize their capacityto extract features in the audio by way of their interaction with thefluid, where a fourth set of artificial neural networks is trained onnon-ray traced or path-traced audio samples and ray-traced orpath-traced audio samples in isolation, where the fourth set ofartificial neural networks receives as input newly arriving ray-tracedor path-traced audio samples arriving at the eardrum location and itsoutput is used to provide baselines and error feedback when training thefirst, second and third sets of artificial neural networks separatelyand in combination, where the outputs from the first set of artificialneural networks is the input to the sensors, interaction activates acorresponding pain effect according to the following table:

Output from the first set of artificial neural networks corresponds todecibel ranges. Sound pain effect. 0.0 to 80 None decibels 80.01 to 130Attenuate down by 0.5 percent per second the decibels resolution of thesensors of sensory type “sound” in the activated sensory locations. Ifthe activated sensory locations become inactive attenuate up by 0.5percent per second the previously attenuated down sensors until theyagain are at their maximum. The maximum can be less than 100 percent oftheir original resolution if the maximum has been lowered 130.01 to 150Temporarily attenuate down by 1 percent per second decibels theresolution of the sensors of sensory type “sound” in the activatedsensory locations and permanently reduce the maximum resolution by 0.1percent per second. If the activated sensory locations become inactiveattenuate up by 1 percent per second the previously attenuated downsensors until they again are at their maximum. The maximum can be lessthan 100 percent of their original resolution if the maximum has beenlowered. Above 150 Temporarily attenuate down by 2 percent per seconddecibels the resolution of the sensors of sensory type “sound” in theactivated sensory locations and permanently reduce the maximumresolution by 0.2 percent per second. If the activated sensory locationsbecome inactive attenuate up by 2 percent per second the previouslyattenuated down sensors until they again are at their maximum. Themaximum can be less than 100 percent of their original resolution if themaximum has been lowered.

An alternate example cause for pain is, in accordance with interactionsbetween sensors in sensory locations representing the virtual being'sears and sense properties in sense locations on the same virtual beingor another virtual being or object, where sensory type of both sensorsand sense properties is sound, where sense locations with sensory typesound emit sound that is ray traced or path traced in the virtualenvironment, where sensors are receiving input from one or morerespective virtual mediums, where the virtual mediums are sound raytracing or path traced audio from the sense locations with sensory typesound arriving at the virtual being's virtualized ear location, wheresensory locations of sensory type sound on the ear location, where thetemporal and spatial characteristics of the ray traced or path tracedaudio arriving at the ear location is mapped to the inputs of thesensory locations of sensory type sound, the interaction activates arespective pain effect according to the following table:

Decibel ranges of sound ray traced or path traced audio arriving at earlocation. Sound pain effect. 0.0 to 80 None decibels 80.01 to 130Attenuate down by 0.5 percent per second the decibels resolution of thesensors of sensory type “sound” in the activated sensory locations. Ifthe activated sensory locations become inactive attenuate up by 0.5percent per second the previously attenuated down sensors until theyagain are at their maximum. The maximum can be less than 100 percent oftheir original resolution if the maximum has been lowered. 130.01 to 150Temporarily attenuate down by 1 percent per second decibels theresolution of the sensors of sensory type “sound” in the activatedsensory locations and permanently reduce the maximum resolution by 0.1percent per second. If the activated sensory locations become inactiveattenuate up by 1 percent per second the previously attenuated downsensors until they again are at their maximum. The maximum can be lessthan 100 percent of their original resolution if the maximum has beenlowered. Above 150 Temporarily attenuate down by 2 percent per seconddecibels the resolution of the sensors of sensory type “sound” in theactivated sensory locations and permanently reduce the maximumresolution by 0.2 percent per second. If the activated sensory locationsbecome inactive attenuate up by 2 percent per second the previouslyattenuated down sensors until they again are at their maximum. Themaximum can be less than 100 percent of their original resolution if themaximum has been lowered.

Another example cause for pain is, in accordance with interactionsbetween sensors in sensory locations representing the virtual vestibularcanals of the virtual being's ears filled with virtual fluid, and senseproperties in sense locations in the virtual fluid or particlescomprising or otherwise within the virtual fluid, where sensory type ofboth sensors and sense properties is balance, where sensors arereceiving input from a respective virtual mediums, one for each of thethree canals, where the virtual mediums model accordingly the virtualfluid so that it is affected by virtual gravity, where the virtualizedfluid is near to or in contact with thousands of virtualized tiny hairsvirtual vestibular canals, where a set of virtual cameras have the hairswithin their viewing frustums and capture and render out frame data,where the frame data is fed into a first set of artificial neuralnetworks training them to recognize the features of individual hairs andclassify their respective movements in the fluid and tensions accordingto change in orientation of the virtual vestibular canals with respectto the movement of vestibular canals occurring when the virtual beingmoves and with respect to virtual gravity effects, where a second set ofartificial neural networks is used to modify and optimize the shape,placement, size and distribution of hairs in the virtual vestibularcanals to optimize their capacity to extract positional features by wayof their interaction with the fluid, where actual positional data fromthe virtual environment engine is used to provide baselines and errorfeedback when training the first and second sets of artificial neuralnetworks, where the outputs from the first set of artificial neuralnetworks is the input to the sensors, the interactions activates arespective pain effect according to the following table:

Continuous rotational spinning Balance pain effect. 0 to 1 rotationsNone in 5 seconds Greater than Attenuate down by 2 percent per secondthe resolution 1 rotation of the sensors of sensory type “balance” inthe every activated sensory locations. If the activated sensory 5seconds locations become inactive attenuate up by 2 percent per secondthe previously attenuated down sensors until they again are at theirmaximum. Store the most recent 10 seconds of input to the activatedsensors, if spinning has occurred for more than 10 seconds and decreasesat a rate of greater than double the rate of increase, replay the mostrecent stored 10 seconds of input as new input to the activated sensors.

In some implementations, the pain model 508 for the AI programs includeperforming one or more operations that include:

-   -   temporarily or permanently reducing speed of computation by        adding delay or reducing clock rate, swapping out an algorithm        implementation for a less efficient implementation, reducing        threads, reducing processes, and reducing computational        resources and/or memory resources;    -   temporarily or permanently reducing accuracy of computation and        associated data;    -   temporarily or permanently removing or changing machine learning        model parameters, optimizer hyper-parameters, and model specific        hyper-parameters;    -   temporarily or permanently removing or changing values stored in        memory for instances of programming language types;    -   temporarily or permanently removing neural nets or layers of a        neural nets; and/or    -   temporarily or permanently removing neural net connections.

In some implementations, the pleasure model 510 for the AI programsinclude performing one or more operations that include:

-   -   temporarily or permanently increasing speed of computation by        removing a delay or increasing clock rate, swapping out an        algorithm implementation for a more efficient implementation,        increasing threads, increasing processes, and increasing        computational, and/or memory resources;    -   temporarily or permanently increasing accuracy of computation        and associated data;    -   temporarily or permanently adding or changing machine learning        model parameters, optimizer hyper-parameters, and model specific        hyper-parameters;    -   temporarily or permanently adding new or re-adding previously        removed or omitted values stored in memory for instances of        programming language types;    -   temporarily or permanently adding or re-adding neural nets or        layers of neural nets; and    -   temporarily or permanently re-adding neural net connections.

In some implementations, the pain model 508 for programs that operate ontype streams include performing one or more operations that include:

-   -   temporarily or permanently reducing speed of computation by        adding delay or reducing clock rate, swapping out an algorithm        implementation for a less efficient implementation, reducing        threads, reducing processes, and reducing computational        resources, memory resources, and/or queue or buffer depth;    -   removing values or instances of a program language from the one        or more type streams; and    -   temporarily or permanently reducing select programs or        operations being applied to the one or more type streams.

In some implementations, the pleasure model 510 for the programs thatoperate on type streams include:

-   -   temporarily or permanently increasing speed of computation by        removing a delay or increasing clock rate, swapping out an        algorithm implementation for a more efficient implementation,        increasing threads, increasing processes, and increasing        computational resources, memory resources, and/or queue or        buffer depth; and    -   temporarily or permanently adding or re-adding select programs        or operations applied to the one or more type streams.

In some implementations, the pain model 512 for the points ofobservation 104 include performing one or more operations that include:

-   -   temporarily or permanently attenuating lower dynamic range        and/or resolution of the one or more points of observation for        the one or more sense properties with matching type descriptors;    -   temporarily or permanently attenuating lower the dynamic range        and/or resolution of the one or more points of observation        having type descriptors matching those of the one or more sense        properties. In some implementations, the modulation effect can        be specified for and/or affect the one or more points of        observation in one or more sensory locations; and    -   temporarily or permanently attenuating lower the dynamic range        and/or resolution of the one or more points of observation        having type descriptors matching and also not matching that of        the medium of observation for one single sensory type that is        imposing a pain effect. In some implementations, the modulation        effect can be specified for and/or affect the one or more points        of observation in one or more sensory locations. In some        implementations, the medium of observation may be for more than        one sense property sensory type and matching or not matching        sensor sensory types.

In some implementations, the pleasure model 514 for the points ofobservation 104 include performing one or more operations that include:

-   -   temporarily or permanently attenuating higher dynamic range        and/or resolution of the one or more points of observation for        the one or more sense properties with matching type descriptors;        and    -   temporarily or permanently attenuating higher the dynamic range        and/or resolution of the one or more points of observation        having type descriptors matching those of the one or more sense        properties. In some implementations, the modulation effect can        be specified for and/or affect the one or more points of        observation in one or more sensory location; and    -   temporarily or permanently attenuating higher the dynamic range        and/or resolution of the one or more points of observation        having type descriptors matching and also not matching that of        the medium of observation for one single sensory type that is        creating a pleasure effect. In some implementations, the        modulation effect can be specified for and/or affect the one or        more points of observation in one or more sensory locations. In        some implementations, the medium of observation may be for more        than one sense property sensory type and matching or not        matching sensor sensory types.

FIG. 6 provides a flowchart of a process 600 for simulating sense datafor virtual beings in virtual environments, according to someimplementations. The method is performed (602) at a computer system(e.g., the computing device 200) having one or more processors (e.g.,the processors 202), memory (e.g., the memory 210), and one or moreprograms stored in the memory and configured for execution by the one ormore processors. The one or more programs include instructions forperforming the method.

The method includes defining (604) a first virtual being (e.g., thevirtual being 110) in a virtual environment (e.g., the virtualenvironment 220). The first virtual being includes a plurality ofsensory locations. Each sensory location has one or more sensors. Eachsensor has a respective sensory type and is configured to receivesensory input by one or more respective virtual mediums having therespective sensory type. The first virtual being also includes aplurality of sense locations, each sense location storing a respectiveset of one or more sense properties, each sense property having arespective sensory type. The first virtual being also includes aplurality of artificial neural networks connecting sensors at theplurality of sensory locations. Each of the artificial neural networkscan be implemented in a variety of ways, such as a simple neuralnetwork, a recurrent neural network, a bidirectional recurrent neuralnetwork, a convolutional neural network, a deep convolutional neuralnetwork, or one of many other known types of neural networks. Anartificial neural network may also be implemented using other machinelearning algorithms, such as a support vector machine, a random forestof decision trees, linear regression, logistic regression, naïve Bayes,kNN, or K-Means.

In some implementations, the method includes defining the sensors andsense properties and of respective types, and/or placing the sensors inthe sensory locations and sense locations, thereby defining spatialcoordinates (sometimes referred to as world coordinate space or localcoordinate space, by different game engines or tools) in a 3D space. Insome implementations, as the first virtual being moves, coordinates areupdated dynamically, and/or the virtual being's body is deformedaccording to movement and/or interactions.

The method also includes defining (606) an object (e.g., an object ofthe one or more objects or other virtual beings 112) in the virtualenvironment. The object has a plurality of sense locations, each senselocation storing a respective set of one or more sense properties, eachsense property having a respective sensory type. It is noted that theobject 118 may include objects and other virtual beings (distinct fromthe virtual being), according to some implementations.

The method also includes, in accordance with an interaction between thevirtual being and the object, receiving (608) sensory input at a firstsensor at a first sensory location using a first virtual mediumaccording to a first sense property of the object at a first senselocation. The first sensor, the first virtual medium, and the firstsense property have a same sensory type. It is noted that, in someimplementations, the first virtual medium determines the interactionbetween sense properties and sensors.

The method also includes, in accordance with the received sensory input,using a first artificial neural network to translate (610) the receivedsensory input into (i) updates to one or more configuration parametersof sensors of the first virtual being and/or (ii) movement of thevirtual being. The artificial neural network or the virtual medium orcombination can update the configuration parameters (e.g. resolution orpruning of sensors when their resolution or max resolution goes to 0 infor some sensory types where it is a permanent pain effect that we want.The artificial neural network also can also be responsible for actuatingthe movement of the being.

Suppose the sensory type is heat. There may be, at a given location onthe virtual being, both heat sensors as well as heat sense properties(e.g., a human arm can sense heat, but it is also at some temperatureitself). As a ball approaches, the sensors can detect the heat, but theheat from the ball will also affect the heat sense properties of thebeing (e.g., raising or lowering the temperature of the virtual being atthe point of contact). In addition, under the pain/pleasure model, thesensors can provide input into the neural networks to modify sensitivityor other parameters of nearby sensors. In some implementations, theupdates include attenuations to parameters like resolution, trainedartificial neural network or machine learning model parameters, and/oralgorithmic parameters. In some implementations, the virtual beinginteracts with itself or another virtual being. In some implementations,the object is a portion of the virtual being, and the virtual beinginteracts with a portion of itself. For example, the virtual being cantouch itself thousands of times per day, and in doing so, feel its ownbody. To further illustrate, similar to a human, the virtual being canlightly touch, rub, stroke, press, or scratch any part of its body usingfingers, touch a tongue to an arm or lick lips, suck a thumb, or runfingers through hair.

In some implementations, the method further includes: selectingenhancements or impairments to apply to the first sensory locationand/or the first artificial neural network based on (i) the firstsensory location and the first sense location, and (ii) the sensory typeof the first sensor, the first virtual medium, and the first senseproperty; and altering, based on the enhancements or impairments, (i)one or more configuration parameters of the first sensor, therebymodifying its ability to receive sensory input, and/or (ii) one or moreconfiguration parameters of the first artificial neural network, therebymodifying its ability to process sense properties. Some implementationsselect a modulation or attenuation effect, and the selection is based ontangible results about the interaction.

In some implementations, in accordance with a determination that thesensory type is light touch, selecting the enhancements or impairmentsis further based on determining whether the first sensory location isdeformed by a predetermined amount in a direction normal to the firstsensory location's non-deformed baseline location, within apredetermined time period.

In some implementations, altering the one or more configurationparameters of the first sensor includes enhancing the ability of thefirst sensor to receive sensory input at the first sensory location byincreasing resolution of the first sensor by a predetermined amount.

In some implementations, the method further includes: in accordance witha determination that the sensory type is temperature, selecting theenhancements or impairments further based on determining whether adistance between the first sensory location and the first sense locationis less than a predetermined amount in a direction normal to the firstsensory location's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes enhancing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by increasing resolution of the first sensor by a firstpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a predetermined minimum temperature up to apredetermined maximum temperature.

In some implementations, selecting the enhancements or impairments isfurther based on determining if the first sensory location is deformedby a predetermined amount in a direction normal to the first sensorylocation's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes enhancing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by increasing resolution of the first sensor by a secondpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a predetermined minimum temperature up to apredetermined maximum temperature.

In some implementations, the method further includes: after altering theone or more configuration parameters of the first sensor and/or the oneor more configuration parameters of the first artificial neural network:in accordance with a determination that the ability of the first sensorto receive sensory input, and/or the ability of the first artificialneural network to sense properties has not changed within apredetermined time period, impairing the ability of one or more sensorsto receive sensory input by decreasing resolution of the first sensor bya predetermined amount. The one or more sensors correspond to sensorylocations with a sensory type of light touch.

In some implementations, in accordance with a determination that thesensory type is pressure, selecting the enhancements or impairments isfurther based on determining whether the first sensory location isdeformed by greater than a predetermined amount in a direction normal tothe first sensory location's non-deformed baseline location, within apredetermined time period.

In some implementations, altering the one or more configurationparameters of the first sensor includes impairing the ability of thefirst sensor to receive sensory input at the first sensory location bydecreasing resolution of one or more sensors, at or near the firstsensory, for a predetermined time period. The one or more sensorscorrespond to sensory locations with a sensory type of light touch orpressure.

In some implementations, the method further includes, in accordance witha determination that resolution of a subset of sensors of the one ormore sensors equals 0, or has a maximum resolution parameter equal to 0,removing the subset of sensors from the first virtual being.

In some implementations, in accordance with a determination that thesensory type is temperature, selecting the enhancements or impairmentsis further based on determining whether a distance between the firstsensory location and the first sense location is less than apredetermined amount in a direction normal to the first sensorylocation's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes impairing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by decreasing resolution of the first sensor by a secondpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a first predetermined minimum temperature or below asecond predetermined maximum temperature.

In some implementations, in accordance with a determination that thesensory type is temperature, selecting the enhancements or impairmentsis further based on determining whether the first sensory location isdeformed by a predetermined amount in a direction normal to the firstsensory location's non-deformed baseline location.

In some implementations, altering the one or more configurationparameters of the first sensor includes impairing the ability of one ormore sensors to receive sensory input, at or near the first sensorylocation, by decreasing resolution of the first sensor by a thirdpredetermined amount for each degree of temperature, as sensed by thefirst sensor, above a predetermined minimum temperature or below afourth predetermined maximum temperature.

In some implementations, the method further includes displaying thefirst virtual being on one or more displays of the computer system.

In some implementations, the object is a second virtual being, distinctfrom the first virtual being.

In some implementations, each sensory location is a point, aone-dimensional segment, a two dimensional area, or a three dimensionalregion.

In some implementations, each sensory location is an n-dimensionalmanifold in the virtual environment, with n=0, 1, 2, or 3.

In some implementations, each sense location is an n-dimensionalmanifold in the virtual environment, with n=0, 1, 2, or 3.

In some implementations, each sensory location corresponds to arespective region, surface, or point, on or within the first virtualbeing.

In some implementations, the plurality of sensory locations aredynamically generated and associated with specific points and/or areason the first virtual being when a surface topology of the first virtualbeing is changing. In some implementations, the sensory locations changebecause the surface topology is changing, but the sensors will remainthe same (e.g., similar to a nerve in a human skin). Someimplementations use a model where, as a reward, new sensory locations(or sensory points) are added. In some implementations, number ofsensors do not increase, although it may decrease because some sensorsare removed. In some implementations, location of sensors in space maychange, but location of the sensors on the manifold remain constant. Forexample, when a skin gets pressed, the location of the nerves in thespace may change, but the skin's manifold relative positions and that ofthe nerves remain constant.

In some implementations, the sensory type includes one or more of:temperature, light touch, pressure, vibration, stretch/compress, sound,and bright.

In some implementations, the method further includes: providing one ormore Application Programming Interface (API) calls to update theplurality of sensory locations; and, in response to receiving a call tothe one or more Application Programming Interface (API) calls,performing one or more operations selected from the group consisting of:creating, writing, reading, modifying, moving, and/or deleting a sensorylocation.

In some implementations, the first virtual being comprises a virtualapproximation of a body of human, an animal, an insect, a humanoid, or acreature.

The terminology used in the description of the invention herein is forthe purpose of describing particular implementations only and is notintended to be limiting of the invention. As used in the description ofthe invention and the appended claims, the singular forms “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will also be understood that theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, steps, operations, elements, and/or components, but donot preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theimplementations were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious implementations with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method of simulating sense data for virtualbeings in virtual environments, performed at a computer system havingone or more processors, memory, and one or more programs stored in thememory and configured for execution by the one or more processors, theone or more programs comprising instructions for: defining a firstvirtual being in a virtual environment, wherein the first virtual beingincludes: a plurality of virtual sensory locations, each virtual sensorylocation having one or more virtual sensors, wherein each virtual sensorhas a respective sensory type and is configured to receive sensory inputby one or more respective virtual mediums having the respective sensorytype; a plurality of virtual sense property locations, each virtualsense property location associated with a respective set of one or moresense properties, each sense property having a respective sensory type;and a plurality of artificial neural networks processing output fromvirtual sensors at the plurality of virtual sensory locations; defininga virtual object in the virtual environment, wherein the virtual objecthas a plurality of virtual sense property locations, each virtual senseproperty location associated with a respective set of one or more senseproperties, each sense property having a respective sensory type; inaccordance with an interaction between the virtual being and the virtualobject, receiving sensory input at a first virtual sensor at a firstvirtual sensory location using a first virtual medium according to afirst sense property of the virtual object at a first virtual senseproperty location, wherein the first virtual sensor, the first virtualmedium, and the first sense property have a same sensory type; and inaccordance with the received sensory input, using a first artificialneural network to translate the received sensory input into (i) updatesto one or more configuration parameters corresponding to resolutionand/or dynamic range of virtual sensors of the first virtual beingand/or (ii) movement of the virtual being, wherein the one or moreconfiguration parameters corresponding to the resolution and/or dynamicrange include one or more hyper-parameters of the first artificialneural network, and updating the hyper-parameters modifies ability ofthe first virtual sensor and/or the first artificial neural network toprocess sensory input from the first virtual medium when there is aninteraction with the first virtual sensor using the first senseproperty.
 2. The method of claim 1, further comprising: selectingchanges to configuration parameters to apply to the first virtualsensory location and/or the first artificial neural network based on (i)the first virtual sensory location and the first virtual sense propertylocation, and (ii) the sensory type of the first virtual sensor, thefirst virtual medium, and the first sense property; and altering, basedon the selected changes to configuration parameters, (i) one or moreconfiguration parameters of the first virtual sensor, thereby modifyingits ability to receive sensory input, and/or (ii) one or moreconfiguration parameters of the first artificial neural network, therebymodifying ability of the first virtual sensor and/or the firstartificial neural network to process sensory input from the firstvirtual medium that determines when there is an interaction with thefirst virtual sensor using the first sense property.
 3. The method ofclaim 2, wherein: in accordance with a determination that the sensorytype is light touch, selecting the changes to configuration parametersis further based on determining whether the first virtual sensorylocation is deformed by a predetermined amount in a direction normal tothe first virtual sensory location's non-deformed baseline location,within a predetermined time period.
 4. The method of claim 3, wherein:altering the one or more configuration parameters of the first virtualsensor comprises enhancing the ability of the first virtual sensor toreceive sensory input at the first virtual sensory location byincreasing resolution of the first virtual sensor by a predeterminedamount.
 5. The method of claim 2, wherein: in accordance with adetermination that the sensory type is temperature, selecting thechanges to configuration parameters is further based on determiningwhether a distance between the first virtual sensory location and thefirst virtual sense property location is less than a predeterminedamount in a direction normal to the first virtual sensory location'snon-deformed baseline location.
 6. The method of claim 5, wherein:altering the one or more configuration parameters of the first virtualsensor comprises enhancing the ability of one or more virtual sensors toreceive sensory input, at or near the first virtual sensory location, byincreasing resolution of the first virtual sensor by a firstpredetermined amount for each degree of temperature, as sensed by thefirst virtual sensor, above a predetermined minimum temperature up to apredetermined maximum temperature.
 7. The method of claim 5, wherein:selecting the changes to configuration parameters is further based ondetermining if the first virtual sensory location is deformed by apredetermined amount in a direction normal to the first virtual sensorylocation's non-deformed baseline location.
 8. The method of claim 7,wherein: altering the one or more configuration parameters of the firstvirtual sensor comprises enhancing the ability of one or more virtualsensors to receive sensory input, at or near the first virtual sensorylocation, by increasing resolution of the first virtual sensor by asecond predetermined amount for each degree of temperature, as sensed bythe first virtual sensor, above a predetermined minimum temperature upto a predetermined maximum temperature.
 9. The method of claim 2,further comprising: after altering the one or more configurationparameters of the first virtual sensor and/or the one or moreconfiguration parameters of the first artificial neural network: inaccordance with a determination that the ability of the first virtualsensor to receive sensory input, and/or the ability of the firstartificial neural network to sense properties has not changed within apredetermined time period, impairing the ability of one or more virtualsensors to receive sensory input by decreasing resolution of the firstvirtual sensor by a predetermined amount, wherein the one or morevirtual sensors correspond to virtual sensory locations with a sensorytype of light touch.
 10. The method of claim 2, wherein: in accordancewith a determination that the sensory type is pressure, selecting thechanges to configuration parameters is further based on determiningwhether the first sensory location is deformed by greater than apredetermined amount in a direction normal to the first virtual sensorylocation's non-deformed baseline location, within a predetermined timeperiod.
 11. The method of claim 10, wherein: altering the one or moreconfiguration parameters of the first virtual sensor comprises impairingthe ability of the first virtual sensor to receive sensory input at thefirst virtual sensory location by decreasing resolution of one or morevirtual sensors, at or near the first virtual sensory location, for apredetermined time period, wherein the one or more virtual sensorscorrespond to virtual sensory locations with a sensory type of lighttouch or pressure.
 12. The method of claim 1, further comprising: inaccordance with a determination that resolution of a subset of virtualsensors of the one or more virtual sensors equals 0, removing the subsetof virtual sensors from the first virtual being.
 13. The method of claim5, wherein: altering the one or more configuration parameters of thefirst virtual sensor comprises impairing the ability of one or morevirtual sensors to receive sensory input, at or near the first virtualsensory location, by decreasing resolution of the first virtual sensorby a second predetermined amount for each degree of temperature, assensed by the first virtual sensor, above a predetermined minimumtemperature or below a predetermined maximum temperature.
 14. The methodof claim 7, wherein: altering the one or more configuration parametersof the first virtual sensor comprises impairing the ability of one ormore virtual sensors to receive sensory input, at or near the firstvirtual sensory location, by decreasing resolution of the first virtualsensor by a third predetermined amount for each degree of temperature,as sensed by the first virtual sensor, above a predetermined minimumtemperature or below a predetermined maximum temperature.
 15. The methodof claim 2, wherein: in accordance with a determination that the sensorytype is sound, selecting the changes to configuration parameters isfurther based on determining whether a decibel range of virtual soundarriving at the first virtual sensory location is within a predeterminedrange; and in accordance with a determination that the sensory type isbright, selecting the changes to configuration parameters is furtherbased on determining whether a duration for which the first virtualsensory location receives bright input is within a first predeterminedtime period.
 16. The method of claim 15, wherein: altering the one ormore configuration parameters of the first virtual sensor comprises:impairing the ability of the first virtual sensor to receive sensoryinput at the first virtual sensory location by decreasing resolution ofone or more virtual sensors, at or near the first virtual sensorylocation, at a predetermined rate, for a second predetermined timeperiod, wherein the one or more virtual sensors correspond to virtualsensory locations with a sensory type of sound or bright; and inaccordance with a determination that the one or more virtual sensorshave become inactive, after the second predetermined time period,increasing the resolution of the one or more virtual sensors until theresolution reaches a predetermined maximum resolution.
 17. The method ofclaim 1, further comprising displaying the first virtual being on one ormore displays of the computer system.
 18. The method of claim 1, whereinthe virtual object is a second virtual being, distinct from the firstvirtual being.
 19. The method of claim 1, wherein each virtual sensorylocation is a point, a one-dimensional segment, a two-dimensional area,or a three-dimensional region.
 20. The method of claim 1, wherein eachvirtual sensory location is an n-dimensional manifold in the virtualenvironment, with n=0, 1, 2, or
 3. 21. The method of claim 1, whereineach virtual sense property location is an n-dimensional manifold in thevirtual environment, with n=0, 1, 2, or
 3. 22. The method of claim 1,wherein each virtual sensory location corresponds to a respectiveregion, surface, or point, on or within the first virtual being.
 23. Themethod of claim 1, wherein the plurality of virtual sensory locationsare dynamically generated and associated with specific points and/orareas on the first virtual being when a surface topology of the firstvirtual being is changing.
 24. The method of claim 1, wherein thesensory type includes one or more of: temperature, light touch,pressure, vibration, stretch/compress, sound, taste, and bright.
 25. Themethod of claim 1, further comprising: providing one or more ApplicationProgramming Interface (API) functions to update the plurality of virtualsensory locations; and in response to receiving a call to the one ormore Application Programming Interface (API) functions, performing oneor more operations selected from the group consisting of: creating,writing, reading, modifying, moving, and/or deleting a virtual sensorylocation.
 26. The method of claim 1, wherein the first virtual beingcomprises a virtual approximation of a body of human, an animal, aninsect, a humanoid, or a creature.
 27. A virtual reality system forsimulating sense data for virtual beings in virtual environments, thesystem comprising: one or more processors; memory; and one or moreprograms stored in the memory and configured for execution by the one ormore processors, the one or more programs comprising instructions for:defining a first virtual being in a virtual environment, wherein thefirst virtual being includes: a plurality of virtual sensory locations,each virtual sensory location having one or more virtual sensors,wherein each virtual sensor has a respective sensory type and isconfigured to receive sensory input by one or more respective virtualmediums having the respective sensory type; a plurality of virtual senseproperty locations, each virtual sense property location associated witha respective set of one or more sense properties, each sense propertyhaving a respective sensory type; and a plurality of artificial neuralnetworks processing output from virtual sensors at the plurality ofvirtual sensory locations; defining a virtual object in the virtualenvironment, wherein the virtual object has a plurality of senseproperty locations, each sense property location associated with arespective set of one or more sense properties, each sense propertyhaving a respective sensory type; in accordance with an interactionbetween the virtual being and the virtual object, receiving sensoryinput at a first virtual sensor at a first virtual sensory locationusing a first virtual medium according to a first sense property of thevirtual object at a first virtual sense property location, wherein thefirst virtual sensor, the first virtual medium, and the first senseproperty have a same sensory type; and in accordance with the receivedsensory input, using a first artificial neural network to translate thereceived sensory input into (i) updates to one or more configurationparameters corresponding to resolution and/or dynamic range of virtualsensors of the first virtual being and/or (ii) movement of the virtualbeing, wherein the one or more configuration parameters corresponding tothe resolution and/or dynamic range include one or more hyper-parametersof the first artificial neural network, and updating thehyper-parameters modifies ability of the first virtual sensor and/or thefirst artificial neural network to process sensory input from the firstvirtual medium when there is an interaction with the first virtualsensor using the first sense property.
 28. A non-transitorycomputer-readable storage medium storing one or more programs configuredfor execution by a computer system having one or more processors andmemory, the one or more programs comprising instructions for: defining afirst virtual being in a virtual environment, wherein the first virtualbeing includes: a plurality of virtual sensory locations, each virtualsensory location having one or more virtual sensors, wherein eachvirtual sensor has a respective sensory type and is configured toreceive sensory input by one or more respective virtual mediums havingthe respective sensory type; a plurality of virtual sense propertylocations, each virtual sense property location associated with arespective set of one or more sense properties, each sense propertyhaving a respective sensory type; and a plurality of artificial neuralnetworks processing output from virtual sensors at the plurality ofvirtual sensory locations; defining a virtual object in the virtualenvironment, wherein the virtual object has a plurality of virtual senseproperty locations, each virtual sense property location associated witha respective set of one or more sense properties, each sense propertyhaving a respective sensory type; in accordance with an interactionbetween the virtual being and the virtual object, receiving sensoryinput at a first virtual sensor at a first virtual sensory locationusing a first virtual medium according to a first sense property of thevirtual object at a first virtual sense property location, wherein thefirst virtual sensor, the first virtual medium, and the first senseproperty have a same sensory type; and in accordance with the receivedsensory input, using a first artificial neural network to translate thereceived sensory input into (i) updates to one or more configurationparameters corresponding to resolution and/or dynamic range of virtualsensors of the first virtual being and/or (ii) movement of the virtualbeing, wherein the one or more configuration parameters corresponding tothe resolution and/or dynamic range include one or more hyper-parametersof the first artificial neural network, and updating thehyper-parameters modifies ability of the first virtual sensor and/or thefirst artificial neural network to process sensory input from the firstvirtual medium when there is an interaction with the first virtualsensor using the first sense property.
 29. The method of claim 2,wherein: in accordance with a determination that the sensory type islight touch, pressure, vibration, or temperature, selecting the changesto configuration parameters is further based on determining whether thefirst virtual sensory location is deformed by more than a predeterminedamount in a specified time interval.
 30. The method of claim 1, whereineach virtual sense property location is a point, a one-dimensionalsegment, a two-dimensional area, or a three-dimensional region.